
p 

i Pate 



United States Patent and Trademark Office 



UNITED STATES DEPARTMENT OF COMMERCE 
United States Patent and Trademark Office 

Address: COMMISSIONER FOR PATENTS 
P.O. Box 1450 

Alexandria, Virginia 223 13-1450 
www.uspto.gdv 



APPLICATION NO. 



FILING 



DATE 



FIRST NAMED INVENTOR 



ATTORNEY DOCKET H 0i " * CONFIRMATION NO. 



10/056,562 



01/25/2002 



7S90 



07/08/^004. 

MarkS. Svat 1 

Fay, Sharpe, Fagan, 

Minnich & McKee, LLP 

1 100 Superior Avenue, 7th Floor 

Cleveland, OH 44114-2518 



Eric Saund 



D/A1318Q,XER2 024§9 



8881 



EXAMINER 



CUNNINGHAM; GREGORY F 



ART UNIT 



PAPER NUMBER 



* 2676 
DATE MAILED: 07/08/2004 



5 



Please find below and/or attached an Office communication concerning this application or proceeding. 



PTO 90C (Rev 10/03) 



#- 



Office Action Summary 



Application No. 

10/056,562 


Applicant(s) 

SAUNDET AL 


Examiner 

Greg Cunningham 


Art Unit 

2676 





- The MAILING DATE of this communication appears on the cover sheet with the correspondence address - 
Period for Reply 

A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) FROM 
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DETAILED ACTION 



1. This action is responsive to communications of amendment received 4/27/2003. 

2. The disposition of the claims is as follows: claims 1-23 are pending in the application. 
Claims 1,11 and 14 are independent claims. 

3. When making claim amendments, the applicant is encouraged to consider the references 
in their entireties, including those portions that have not been cited by the examiner and their 
equivalents as they may most broadly and appropriately apply to any particular anticipated claim 
amendments. 



5. The following is a quotation of the appropriate paragraphs of 35 U.S. C. 102 that form the 
basis for the rejections under this section made in this Office action: 

A person shall be entitled to a patent unless - 

(a) the invention was known or used by others in this country, or patented or described in a printed publication in this 
or a foreign country, before the invention thereof by the applicant for a patent 

6. Claims 1-3, 7, 8, 1 1 and 12 are rejected under 35 U.S.C. 102(a) as being disclosed by 
Seni et al., (PGPUB-DOCUMENT-NUMBER: 20030007018), hereafter Seni. 

A. Claim 1, "An image analysis and conversion method comprising: receiving a digital ink 
image 



Specification 



4. 



In view of amended specification, objection is withdrawn. 



Claim Rejections - 35 USC § 102 
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[Often it is difficult to differentiate between these two modes of stylus operation, viz. that of a 
writing implement for text entry (inking mode) and its control function such as for clicking on 
application icons and the like (control mode). - para. 0007; 

The present invention is a method of interfacing with and a handwriting user interface 
(HUI) for small (pocket-shirt sized) portable devices with a touch-enabled input/output (I/O) 
screen, such as are commonly known as personal digital assistants (PDAs). The portable devices 
may be capable of wireless message transmission (such as for web browsing and/or e-mail). The 
user interface of the present invention is typically in software and loaded into PDA storage. A 
state of the art handwriting recognition engine also is included in software. The handwriting user 
interface of the present invention enhances the usability, flexibility and power of the handheld 
device in which it is installed. - para. 0015; 

However, small digital ink point at the end of a word is much easier to identify and 
classify as a punctuation mark, e.g. a period, comma, etc. - para. 0024; 

These errors and conflicts also result from the inherent ambiguity of inputting with a 
single pointing device, i.e., a stylus, wherein the stylus is used both as an inking pen for writing 
and, as a mouse-type pointing device for function selection. For example, the device must 
distinguish between an inking stroke and scrolling the screen by dragging the stylus. By 
designating an input area for writing, such conflicts are resolved simply: the stylus functions as 
an inking pen inside the writing area and as a non-inking pointing device/mouse outside of the 
input area. - para. 0026]; and 

converting the digital ink image into structured object representations of the digital ink 

image 
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[An entire message may be quickly handwritten, converted , stored and 
then, transmitted, for example. - para. 0015], 

which are editable by a structured text/graphics editor 

[FIG. 1 shows a preferred embodiment pocket sized handheld device 100 with a housing 
101 graphical handwriting user interface 102 according to preferred embodiment of the present 
invention, - para. 0016; 

Action icons 106, 108, 1 10, 1 12, 1 14 are displayed to provide virtual buttons for editing 
any previously entered text. Preferably, the icons are displayed together at any side of the input 
area (e.g., left, right, top or bottom). Editing operations may include, but are not limited to: 
insert a space 108, backspace 112, delete 114, capitalize recognition result 1 10, and undo 
insertion of last recognition result 106. Further, as each word is entered and recognized, a stylus 
may be used to select one or more characters of the word in a text field of the active application. 
The preferred recognition engine is also capable of recognizing individual stand-alone 
characters. At any time, the user can select one (or more) character(s) from a previously entered 
word and write a new character(s) in the input area with the result replacing the selected text. - 
para. 0022; 

The previously input text is displayed at the top of the screen. Each word is entered and 
the last recognition result remains displayed for editing in the editing area. As noted above, a 
single word can be selected or, individual letters within the word may be selected and corrected 
using the QWERTY keyboard 132. - para. 0023; 

Typical recognition options may include an option to propose upper-case at the beginning 
of a word, an option to suggest end of word punctuation, the number of recognition results 
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displayed in the pop-up list, the location of editing buttons (i.e., left or right hand side of the 
input area), and user dictionary maintenance, i.e., viewing, adding, and/or deleting entries. The 
option to propose upper-case may be such that, if set, the recognition engine attempts to 
recognize the input with and without a leading upper-case letter. - para. 0024]" is disclosed [as 
detailed]. 

B. Claim 2, "The method according to claim 1 wherein the converting step includes, altering 
the digital ink image into multiple alternative interpretations" [para. 0024], is disclosed supra for 
claim 1 and [as detailed]. Wherein recognition options correspond to alternative interpretations. 

C. Claim 3, "The method according to claim 2 wherein the altering of the digital ink image 
into multiple alternative interpretations includes, altering the digital ink image into informal 
structured object representations that are editable by the structured text/graphics editor; and 
altering the digital ink image into formal structured object representations that are editable by the 
structured text/graphics editor", is disclosed supra for claim 2 and wherein informal is depicted 
at 104 and formal is depicted under 116, "happy" of Fig. 1. 

D. Claim 7, "The method according to claim 1 wherein the step of converting the digital ink 
image to the structured object representations includes generating multiple structured object 
representations of the digital ink image [para. 0016 at "a secondary list of potential recognition 
candidates may be displayed in a box 120"], the multiple structured object representations 
representing at least a first image representation having formal structured object representations 
[para. 0016 at "As each word is recognized, it is shown inserted into the text at the top of the 
interface display 102], and a second image representation containing informal structured object 
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representations [para. 0016 at "A lower portion of the display is designated as a handwriting 
input area 104."] is disclosed supra for claim 1 and [as detailed]. 

E. Claim 8, "The method according to claim 1 wherein the editing by the structured 
text/graphics editor permits movement of structured object representations by at least one of, 
individual objects, a sub-group of all the structured object representations [para. 0022 at "Editing 
operations may include, but are not limited to: insert a space 108, backspace 112, delete 114, 
capitalize recognition result 110, and undo insertion of last recognition result 106."], or as an 
overall group of the structured object representations" is disclosed supra for claim 1 and [as 
detailed]. Wherein delete corresponds to editing a sub-group of all the structured object 
representations. Said sub-group of all the structured object representations corresponds a letter or 
character of a word or multi-digit number. 

F. Per independent claim 11, this is directed to a system for performing the method of 
independent claim 1, and therefore is rejected to independent claim 1. 

G. Per dependent claim 12, this is directed to a system for performing the method of claim 1 
and in part of dependent claim 7, and therefore is rejected to claim 1 and dependent claim 7. 

Claim Rejections - 35 USC § 103 
7. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 
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8. Claim 4 is rejected under 35 U.S.C. 103(a) as being unpatentable over Seni as applied to 
claim 1 above, and further in view of Official Notice. 

A. Claim 4, "The method according to claim 1 wherein the step of converting the digital ink 
image into structured object representations of the digital ink image includes configuring the 
structured object representations to represent an electronic slide of the structured text/graphics 
editor " is disclosed supra for claim 1 . Although Seni does not appear to disclose, "includes 
configuring the structured object representations to represent an electronic slide", Official notice 
is taken that the art is replete with importing capabilities to configure structured object 
representations to represent an electronic slides for example Powerpoint by Microsoft 
Corporation. 

Therefore it would have been obvious to one of ordinary skill in the art at the time the 
invention was made to apply digital ink recognition disclosed by Seni in combination with 
electronic slides disclosed by Official Notice (Powerpoint), and motivated to combine the 
teachings because it would be obvious since these features are well known in the art. 

9. Claims 5 and 9 are rejected under 35 U.S.C. 103(a) as being unpatentable over Seni as 
applied to claim 1 above, and further in view of Wilcox et al., (US-PAT-NO: 5,889,523), 
hereafter Wilcox. 

A. Claim 5, "The method according to claim 1, wherein the converting step includes 
forming of an Alternative graph" is disclosed by Seni supra for claim 1 . Although Seni does not 
appear to disclose "wherein the converting step includes forming of an Alternative graph", 
Wilcox does in abstract and col. 2, Ins. 24-46. Wherein cluster tree corresponds to alternative 
graph. 
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Therefore it would have been obvious to one of ordinary skill in the art at the time the 
invention was made to apply digital ink recognition disclosed by Seni in combination with 
cluster tree (alternative graph) disclosed by Wilcox, and motivated to combine the teachings 
because it would it is an important interface issue in graphical editing systems as revealed by 
Wilcox in col. 1, lines 12-16. 

B. Claim 9, "The method according to claim 1 wherein the digital ink image is converted 
into the structured objects representations of the digital ink image through the use of an 
Alternative Graph" is disclosed by Seni supra for claim 1 and Wilcox for claim 5. 
10. Claims 14, 15, 17-20, 22 and 23 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Seni as applied to claim 1 above, and further in view of Golovchinsky et al., 
(US-PAT-NO: 6,389,435), hereafter Golovchinsky. 

A. Claim 14, "On a screen display of an electronic device operating a structured 
text/graphics editor, an image representation comprising: structured object representations of a 
digital ink image, the structured object representations correlating to perceptually salient areas of 
the digital ink image, wherein the structured object representations are editable by the structured 
text/graphics editor to allow a user to generate alternative interpretations of the digital ink 
image" is disclosed by Seni supra for claim 1 . Although Seni does not appear to disclose 
"representations correlating to perceptually salient areas of the digital ink image", Golovchinsky 
does at [An embodiment of the system and method of the invention provides a perceptually- 
motivated model of freeform digital ink marks that applies higher weight to more salientlv 
marked terms. For example, a user may choose to use digital ink that has a high salience such as 
a bright color and the system may apply a higher weight to the terms that are marked with this 
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bright color digital ink than those terms that are marked with a less salient digital ink for the 
query. The search results will reflect the greater weight given to the terms marked with the high 
salience freeform digital ink . - col. 4, Ins. 58-67]. ~ 

Therefore it would have been obvious to one of ordinary skill in the art at the time the 
invention was made to apply digital ink recognition disclosed by Seni in combination with 
salient digital ink disclosed by Golovchinsky, and motivated to combine the teachings because 
combinations of shapes and colors can be used for the freeform digital ink marks to create a large 
number of identifiable terms as revealed by Golovchinsky in col. 2, lines 23-25. 

B. Claim 15, "The image representation according to claim 14 wherein the structured object 
representations are informal structured object representations, and wherein the informal 
structured object representations are editable to formal structured object representations" is 
disclosed supra for claim 14 and furthermore by Seni in [para. 0022], particularly at "At any 
time, the user can select one (or more) character(s) from a previously entered word and write a 
new character(s) in the input area with the result replacing the selected text". 

Therefore it would have been obvious to one of ordinary skill in the art at the time the 
invention was made to apply digital ink recognition disclosed by Seni in combination with 
salient digital ink disclosed by Golovchinsky, and motivated to combine the teachings because 
combinations of shapes and colors can be used for the freeform digital ink marks to create a large 
number of identifiable terms as revealed by Golovchinsky in col. 2, lines 23-25. 

C. Claim 17, "The image representation according to claim 14 wherein a first structured 
object representation is spatially contained within a second structured object representation" is 
disclosed by Seni and Golovchinsky supra for claim 14. Wherein a letter corresponds to a first 
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structured object and a word corresponds to a second structured object wherein the two are 
spatially connected. 

D. Claim 18, "The image representation according to claim 14 wherein a new structured 
object representation is added to existing structured object representations" is disclosed by Seni 
and Golovchinsky supra for claim 14. See Seni, Fig. 1, wherein each new word is added to 
existing structured text at top of area 102. 

E. Claim 19, "The image representation according to claim 14 wherein the structured object 
representations define a text block structure" is disclosed by Seni and Golovchinsky supra for 
claim 14. See Seni, Fig. 1. 

F. Claim 20, "The image representation according to claim 19 wherein the text block 
structure includes a display of text parameters including at least one of text layout, text font, 
bullets, underlines and dummy characters" is disclosed by Seni and Golovchinsky supra for 
claim 18. See Seni, Fig. 1, depicted text layout. 

G. Claim 22, "The image representation according to claim 14 wherein distinct alternative 
interpretations may be displayed at the same time" is disclosed by Seni and Golovchinsky supra 
for claim 14. See Seni, Fig. 1, pop-up window list 120. 

H. Claim 23, "The image representation according to claim 22 wherein display of the 
alternative interpretations is accomplished by the use of at least one of underlays, bubble or 
balloon images, coloring, shading transparency/translucency, defocusing, and pop-up windows" 
is disclosed by Seni and Golovchinsky supra for claim 14. See Seni, Fig. 1, pop-up window list 
120. 
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11. Claim 21 is rejected under 35 U.S.C. 103(a) as being unpatentable over Seni as applied to 
claim 1 above, further in view of Golovchinsky et al, (US-PAT-NO: 6,389,435), hereafter 
Golovchinsky, and further in view of Official Notice. 

A. Claim 21, "The image representation according to claim 20 wherein the dummy 
characters are replaceable with target characters" is disclosed by Seni and Golovchinsky supra 
for claim 20. However they do not appear to disclose, "wherein the dummy characters are 
replaceable with target characters", but Official notice is taken that the art is replete wherein the 
dummy characters are replaceable with target characters in the form of temporary characters, 
templates, boiler plate documents and slides, and place holders. 

Therefore it would have been obvious to one of ordinary skill in the art at the time the 
invention was made to apply digital ink recognition disclosed by Seni in combination with 
salient digital ink disclosed by Golovchinsky coupled with dummy characters disclosed by 
Official Notice (target characters in the form of temporary characters, templates, boiler plate 
documents and slides, and place holders), and motivated to combine the teachings because 
combinations of shapes and colors can be used for the freeform digital ink marks to create a large 
number of identifiable terms as revealed by Golovchinsky in col. 2, lines 23-25. 

12. Claim 16 is rejected under 35 U.S.C. 103(a) as being unpatentable over Seni as applied to 
claim 1 above, further in view of Golovchinsky et al., (US-PAT-NO: 6,389,435), hereafter 
Golovchinsky and further in view of Mahoney et al., (US-PAT-NO: 6,470,095), hereafter 
Mahoney. 

A. Claim 16, "The image representation according to claim 15 wherein the alternative 
interpretations permit a mixing of formal structured object representations and informal 
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structured object representations in a single image representation displayed on the computer 
screen" is disclosed by Seni and Golovchinsky supra for claim 15. Although Seni and 
Golovchinsky do not appear to disclose, "wherein the alternative interpretations permit a mixing 
of formal structured object representations and informal structured object representations in a 
single image representation displayed on the computer screen", Mahoney does in Fig. 3 at col. 2, 
Ins. 45-49. 

Therefore it would have been obvious to one of ordinary skill in the art at the time the 
invention was made to apply digital ink recognition disclosed by Seni in combination with 
salient digital ink disclosed by Golovchinsky coupled with handwritten and typeset text 
displayed in a single image disclosed by Mahoney, and motivated to combine the teachings 
because combinations of shapes and colors can be used for the freeform digital ink marks to 
create a large number of identifiable terms as revealed by Golovchinsky in col. 2, lines 23-25 and 
because the amount of a human user's time required to enter and edit such user-created borders is 
significant as revealed by Mahoney in col. 1, Ins. 22-23. 

Allowable Subject Matter 

13. Claims 6, 10 and 13 are objected to as being dependent upon a rejected base claim, but 
would be allowable if rewritten in independent form including all of the limitations of the base 
claim and any intervening claims. 

Response to Arguments 

14. Contrary to applicants' remarks that Seni is "unrelated to interpreting mixed text/graphic 
digital ink and creating structured representations" wherein actually claim 1 reads as "An image 



Application/Control Number: 10/056,562 Page 13 

Art Unit: 2676 

analysis and conversion method comprising: receiving a digital ink image; and converting the 
digital ink image into structured object representations of the digital ink image, which are 
editable by a structured text/graphics editor." Although the claims are read in light of the 
specification, the specification is not read into the claims, neither is "interpreting mixed 
text/graphics digital ink". However Seni does clarify [handwriting input] as both graphics 
[graphical handwriting user interface 102 - para. 0016] and text [Recognized text is displayed at 
the top of the screen - also in para. 0016]. 

Claim 1 does not distinguish between "structured object representations of the digital ink 
image" and Seni's [graphical handwriting user interface with text recognition], even when focus 
is directed to the instant specification, (on page 6, para. [0035]), where digital ink images are 
defined as symbols (i.e. structured object representations) and not converted into online images. 
Claim 1 does not mention "online images" and even if it did, the applicants' specification simply 
does not elucidate that "online images" are divergent from Seni's [recognized text from the 
graphical handwriting user interface]. 

Although the applicants' specification, as pointed to in para. [0037 - 0038], further 
conveys examples of image editing operations and rudimentary editing operations on picture 
objects, this has not been recited in claim 1. Whereas claim 1 merely reads, "which are editable 
by a structured text/graphics editor". 

In addition claim 1 is not comprehensively sufficient as detailed in applicants' 
specification, para. [0039 - 0040], for movement of the image as illustrated in figure 2C of the 
instant application, and is rather expressly silent for how the "structured text/graphics editor" 
differentiates from Seni's editor as detailed by Seni [in para. 0022]. 
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Independent claim 1 1 is of similar elements as claim 1 and the same reasoning applies as 
applied to claim 1 . 

Seni discloses claim 14, supra for claim 1, and combined with Golovchinsky [col. 4, Ins. 
58-67] to disclose "the structured object representations correlating to perceptually salient areas 
of the digital ink image". Here Golovchinsky discloses [higher weight to more saliently marked 
terms. For example, a user may choose to use digital ink that has a high salience such as a bright 
color and the system may apply a higher weight to the terms that are marked with this bright 
color digital ink than those terms that are marked with a less salient digital ink for the query.], 
which correspond to "the structured object representations correlating to perceptually salient 
areas of the digital ink image" of claim 14. This can be further correlated to: 
[a perceptually-motivated model of freeform digital ink marks] corresponds to "the structured 
object representations" and [higher weight to more saliently marked terms (high salience such as 
a bright color)] corresponds to "perceptually salient areas of the digital ink image". 

Although Golovchinsky uses the phrase [freeform digital ink marks] which one would 
intend implies unstructured, nevertheless must have sufficient structure to be recognized as 
having specific meaning as described by Golovchinsky at [Freeform digital ink marks that are 
not recognized as having a specific meaning will still select a term in the index but will not affect 
the logic of the search. - col. 4, Ins. 54-57, see also Ins. 40-53]. 

Golovchinsky continues in disclosing [Another embodiment of the system and method of 
the invention may be trained to define an ink query grammar that affects how the logic of a query 
is constructed from the index terms and from metadata. For example, the position, shape, type or 
other attribute of a user's marks can affect the formulation of a query. In the absence of a 
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grammar, all selected terms are given equal weight in the query. However, if freeform digital 
ink marks are parsed, the user can specify advanced search features. An example of such an 
advanced feature determines which of the selected terms are mandatory. This example may be 
implemented by putting a freeform digital ink plus a sign, a star, or other mark by a term to 
determine which terms should occur in the results while using other freeform digital ink marks 
such as a cross-out to indicate which terms are not to occur in the results. - col. 4, Ins. 40-54] 
which corresponds to "structured object representations are editable by the structured 
text/graphics editor to allow a user to generate alternative interpretations of the digital ink 
image" 

As far as "a representation for mixed text and graphics in which formal or informal 
representations of text and graphics objects can be swapped in and out by a user at will" is not 
claimed by claim 14. 

Conclusion 

15. Applicant's amendment necessitated the new ground(s) of rejection presented in this 
Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). 
Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within TWO 
MONTHS of the mailing date of this final action and the advisory action is not mailed until after 
the end of the THREE-MONTH shortened statutory period, then the shortened statutory period 
will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 
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CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, 
however, will the statutory period for reply expire later than SIX MONTHS from the date of this 
final action. 

Responses 

16. Responses to this action should be mailed to: Commissioner of Patents and Trademarks, 
Washington, D.C. 20231. If applicant desires to fax a response, (703) 872-9314 may be used for 
formal communications. 

Please label "PROPOSED" or "DRAFT" for informal facsimile communications. Hand- 
delivered responses should be brought to Crystal Park II, 2121 Crystal Drive, Arlington, VA., 
Sixth Floor (Receptionist). 

When making claim amendments, the applicant is encouraged to consider the references 
in their entireties, including those portions that have not been cited by the examiner and their 
equivalents as they may most broadly and appropriately apply to any particular anticipated claim 
amendments. 

Inquiries 

1 7. Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Greg Cunningham whose telephone number is (703) 308-6109. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Matthew Bella, can be reached on (703) 308-6829. 
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Any inquiry of a general nature or relating to the status of this application or proceeding 
should be directed to the receptionist whose telephone number is (703) 305-4700. 




Primary Examiner 



